On the Use and Implementation of Message Logging

Message logging has long been advocated as offering better failure-free performance than coordinated checkpointing. On the contrary, we present a number of experiments showing that for compute-intensive applications executing in parallel on clusters of workstations, message logging has higher failure-free overhead than coordinated checkpointing. Message logging protocols, however, result in much shorter output latency than coordinated checkpointing. Therefore, message logging should be used for applications involving substantial interactions with the outside world, while coordinated checkpointing should be used otherwise. We also present an unorthodox message logging design that uses coordinated checkpointing with message logging, departing from the conventional approaches that use independent checkpointing. This combination of message logging and coordinated checJcpointing offers several advantages, including improved failure-free performance, bounded recovery time, simplified garbage collection, and reduced complexity. Meanwhile, the new protocols retain the advantages of the conventional message logging protocols with respect to output commit. Finally, we discuss three “lessons learned” from an implementation of various message logging protocols. First, during output commit, only the dependency information for the messages in the log needs to be written to the stable storage. It is not necessary to write the message data to stable storage, leading to faster output commit. Second, the use of copy-on-write in the implementation of message logging substantially reduces the logging overhead for communication-intensive programs. Finally, we provide quantitative evidence supporting previous qualitative claims about the superiority of sender-based message logging over receiver-based logging. ‘This work was supported in part by NFS Grants CDA-9222911 and CCR-9116343, and by the Texas Advanced Technology Program Grants ATP 003604012 and ATP 0036041014. The first author was also supported in part by an IBM Graduate Fellowship, and by the Advanced Research Projects Agency under contract number DABT63-93-C-0054. The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies of the sponsors. Willy Zwaenepoel Department of Computer Science Rice University Houston, TX 77251